Implementation testing of a hybrid symbolic/statistical multimodal architecture

The design and implementation of hybrid symbolic/statistical architectures is a major area of interest in current multimodal system development. Such an architecture attempts to improve multimodal recognition and disambiguation rates by using corpus-based statistics to weight the contributions from various input streams. This is in contrast to current architectures that assume independence between input streams, and combine un-weighted posterior probabilities simply by taking their cross product. Recently a Members, Teams, Committee (MTC) approach for statistically hybridizing the Quickset multimodal system has been put forward on the basis of strong empirical results in an offline analysis. MTC uses small-dimensional input streams as Members, which in turn are input into various Teams where their conditional weights are trained. The Committee then extracts a decision from the output of the Teams. This paper discusses a fully implemented regression test of MTC within Quickset, and our modification of the approach to use more specific training features. We report a relative decrease in multimodal error rate of 30%.